Skip to content

Tramp for Lem - #2271

Open
MasterCsquare wants to merge 23 commits into
lem-project:mainfrom
MasterCsquare:tramp
Open

Tramp for Lem#2271
MasterCsquare wants to merge 23 commits into
lem-project:mainfrom
MasterCsquare:tramp

Conversation

@MasterCsquare

@MasterCsquare MasterCsquare commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

lem-tramp

Transparent remote file editing for Lem. Works like
Emacs TRAMP — type a remote path
into C-x C-f and edit the file as if it were local.

Supported Methods

Method Syntax Description
ssh /ssh:user@host:/remote/path Edit files on a remote host via SSH
sudo /sudo::/local/path Edit local files with root privileges via sudo

Usage

Open a file with C-x C-f using the TRAMP path syntax:

C-x C-f /ssh:root@example.com:/etc/nginx/nginx.conf
C-x C-f /sudo::/etc/hostname

The remote file is loaded into a buffer. Edit normally, then save with C-x C-s.

Authentication

SSH (/ssh:)

Key-based auth (recommended) — no prompt, works automatically if you have
SSH keys set up and your key is in the remote host's authorized_keys.

Password auth — requires the sshpass utility:

# Arch
sudo pacman -S sshpass
# Debian/Ubuntu
sudo apt install sshpass

When key auth is unavailable, a password prompt appears in the minibuffer.
The password is cached for the session; subsequent file operations on the
same host reuse it without re-prompting.

Sudo (/sudo::)

A password prompt appears if your sudo timestamp has expired (typically
5-15 minutes since your last sudo invocation in a terminal). If you
have passwordless sudo configured (NOPASSWD in sudoers), no prompt is
shown.

How It Works

lem-tramp hooks into Lem's virtual filesystem layer:

  • Reading — runs cat /remote/path via SSH (or sudo), loads the
    output directly into an in-memory buffer.
  • Writing — pipes the buffer content through cat > /remote/path.
  • Directory listing — uses ls -1a for completion and directory-mode.
  • Metadata — uses stat -c '%s %Y' for file size and modification time.

No temporary files are created on either the local or remote side.

Dependencies

  • sshpass — only needed for SSH password authentication
  • flexi-streams, str, babel, ppcre — Common Lisp libraries

Performance

SSH connections use ControlMaster multiplexing — the first command
establishes the TCP connection, and all subsequent commands reuse it.
A 5-second filesystem cache eliminates redundant test -d / test -f /
stat calls when Lem probes the same path multiple times during a single
file-open operation.

Replaced the unconditional password prompt in the sudo auth path
with a `sudo -n true` pre-check.  On a fresh connection this runs
in ~0.02s, providing a natural UI settling delay — the same reason
SSH never had this problem (its BatchMode connection test takes ~3s).

When `sudo -n true` succeeds (recent sudo timestamp still valid),
no password prompt is needed at all, matching terminal sudo behaviour.
@code-contractor-app

code-contractor-app Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

❌ Code Contractor Validation: FAILED

=== Contract: contract ===

✓ Code Contractor Validation Result
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📋 Contract Source: Repository

📊 Statistics:
  Files Changed:    7
  Lines Added:      695
  Lines Deleted:    32
  Total Changed:    727
  Delete Ratio:     0.04 (4%)

Status: FAILED ❌

🤖 AI Providers:
  - codex — model: (Codex default)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
⚠️ Violations Found (5):

[WARNING] max_total_changed_lines
  Too many lines changed: 727 > 400
  Limit: 400
  Actual: 727

[ERROR] file_structure_rule
  AI check failed: "file_structure_rule"
  ❌ Reason:
    Added `defvar` declarations appear after function definitions in
    `tramp.lisp`, which violates the required top-to-bottom file order.

[ERROR] loop_keywords_rule
  AI check failed: "loop_keywords_rule"
  ❌ Reason:
    An added `loop` form uses unqualified loop keywords instead of the
    required colon-prefixed form.

[ERROR] functional_style_rule
  AI check failed: "functional_style_rule"
  ❌ Reason:
    The change introduces new global dynamic state with `defvar` for
    passwords and caches, instead of passing state explicitly.

[ERROR] macro_style_rule
  AI check failed: "macro_style_rule"
  ❌ Reason:
    The added code uses backquote to construct lists in ordinary functions,
    where this rule requires preferring `list` outside macros.
📋 Contract Configuration: contract (Source: Repository)
version: 2

trigger:
  paths:
    - "extensions/**"
    - "frontends/**/*.lisp"
    - "src/**"
    - "tests/**"
    - "contrib/**"
    - "**/*.asd"
  head_branches:
    exclude:
      - 'revert-*'

validation:
  limits:
    max_total_changed_lines: 400
    max_delete_ratio: 0.5
    max_files_changed: 10
    severity: warning

  ai:
    system_prompt: |
      You are a senior Common Lisp engineer reviewing code for Lem editor.
      Lem is a text editor with multiple frontends (ncurses, SDL2, webview).
      Focus on maintainability, consistency with existing code, and Lem-specific conventions.
    rules:
      # === File Structure ===
      - name: defpackage_rule
        prompt: |
          First form must be `defpackage` or `uiop:define-package`.
          Package name should match filename (e.g., `foo.lisp` → `:lem-ext/foo` or `:lem-foo`).
          Extensions must use `lem-` prefix (e.g., `:lem-python-mode`).

      - name: file_structure_rule
        prompt: |
          File organization (top to bottom):
          1. defpackage
          2. defvar/defparameter declarations
          3. Key bindings (define-key, define-keys)
          4. Class/struct definitions
          5. Functions and commands

          Only flag violations when elements appear OUT OF ORDER (e.g., key bindings AFTER functions, or defvar AFTER functions).
          Key bindings appearing BEFORE functions is correct and expected.

      # === Style ===
      - name: loop_keywords_rule
        prompt: |
          Loop keywords must use colons: `(loop :for x :in list :do ...)`
          NOT: `(loop for x in list do ...)`

      - name: naming_conventions_rule
        prompt: |
          Naming conventions:
          - Functions/variables: kebab-case (e.g., `find-buffer`)
          - Special variables: *earmuffs* (e.g., `*global-keymap*`)
          - Constants: +plus-signs+ (e.g., `+default-tab-size+`)
          - Predicates: -p suffix for functions (e.g., `buffer-modified-p`)
          - Do NOT use -p suffix for user-configurable variables

      # === Documentation ===
      - name: docstring_rule
        prompt: |
          Required docstrings for:
          - Exported functions, methods, classes
          - `define-command` (explain what the command does)
          - Generic functions (`:documentation` option)
          Important functions should explain "why", not just "what".
        severity: warning

      # === Lem-Specific ===
      - name: internal_symbol_rule
        prompt: |
          Use exported symbols from `lem` or `lem-core` package.
          Avoid `lem::internal-symbol` access.
          If internal access is necessary, document why.

      - name: error_handling_rule
        prompt: |
          - `error`: Internal/programming errors
          - `editor-error`: User-facing errors (displayed in echo area)
          Always use `editor-error` for messages shown to users.

      - name: frontend_interface_rule
        prompt: |
          Frontend-specific code must use `lem-if:*` protocol.
          Do not call frontend implementation directly from core.
        severity: warning

      # === Functional Style ===
      - name: functional_style_rule
        prompt: |
          Prefer explicit function arguments over dynamic variables.
          Avoid using `defvar` for state passed between functions.
          Exception: Well-documented cases like `*current-buffer*`.

      - name: dynamic_symbol_call_rule
        prompt: |
          Avoid `uiop:symbol-call`. Rethink architecture instead.
          If unavoidable, document the reason.

      # === Libraries ===
      - name: alexandria_usage_rule
        prompt: |
          Alexandria utilities allowed: `if-let`, `when-let`, `with-gensyms`, etc.
          Avoid: `alexandria:curry` (use explicit lambdas)
          Avoid: `alexandria-2:*` functions not yet used in codebase

      # === Macros ===
      - name: macro_style_rule
        prompt: |
          Keep macros small. For complex logic, use `call-with-*` pattern:
          ```lisp
          (defmacro with-foo (() &body body)
            `(call-with-foo (lambda () ,@body)))
          ```
          Prefer `list` over backquote outside macros.

💬 Feedback

Reply to a violation comment with:

  • /dismiss <reason> - Report false positive or not applicable

🔁 Re-validation

Validation does not re-run automatically when you push fixes. Trigger it with one of:

  • Click Run Validation on the Code Contractor check in the Checks tab
  • Comment @code-contractor-app on this pull request
  • Comment /revalidate to re-run and restore previously dismissed violations
📚 About Code Contractor

Declarative Code Standards That Learn and Improve

Define domain-specific validation rules in YAML.
Your contracts document team knowledge and evolve into more accurate AI enforcement.

Want this for your repo?
Install Code Contractor

@code-contractor-app code-contractor-app Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Contractor validation failed ❌ — see the sticky comment for full results.

Comment thread extensions/tramp/tramp.lisp Outdated
;;; Password Management
;;; ------------------------------------------------------------------

(defvar *tramp-passwords* (make-hash-table :test 'equal)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Contractor: file_structure_rule

Contract: contract

AI check failed: "file_structure_rule"

Reason:
Added defvar declarations appear after function definitions in tramp.lisp, which violates the required top-to-bottom file order.


💬 Reply /dismiss <reason> to dismiss this violation.

Comment thread extensions/tramp/tramp.lisp Outdated
(close in)
(let ((stdout
(with-output-to-string (s)
(loop for line = (read-line out nil nil)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Contractor: loop_keywords_rule

Contract: contract

AI check failed: "loop_keywords_rule"

Reason:
An added loop form uses unqualified loop keywords instead of the required colon-prefixed form.


💬 Reply /dismiss <reason> to dismiss this violation.

Comment thread extensions/tramp/tramp.lisp Outdated
;;; Password Management
;;; ------------------------------------------------------------------

(defvar *tramp-passwords* (make-hash-table :test 'equal)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Contractor: functional_style_rule

Contract: contract

AI check failed: "functional_style_rule"

Reason:
The change introduces new global dynamic state with defvar for passwords and caches, instead of passing state explicitly.


💬 Reply /dismiss <reason> to dismiss this violation.

Comment thread extensions/tramp/tramp.lisp Outdated
(cmd-args (if (listp command) command (list command)))
(control-opts (ssh-control-options)))
(if password
`("sshpass" "-p" ,password

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Contractor: macro_style_rule

Contract: contract

AI check failed: "macro_style_rule"

Reason:
The added code uses backquote to construct lists in ordinary functions, where this rule requires preferring list outside macros.


💬 Reply /dismiss <reason> to dismiss this violation.

@MasterCsquare MasterCsquare changed the title WIP: Tramp Tramp Jul 21, 2026
@MasterCsquare MasterCsquare changed the title Tramp Tramp for Lem Jul 21, 2026
@MasterCsquare MasterCsquare changed the title Tramp for Lem WIP: Tramp for Lem Jul 21, 2026
Two-pronged fix for keystrokes from the sudo password prompt
appearing at the top of the opened buffer:

1. Move the password prompt into the expand-file-name handler, which
   runs first in the file-open pipeline — before any directory probes
   and before the target buffer is created.  This gives the UI a clean
   slate with no buffer to leak into.

2. Add a defensive strip in %read-remote-file: if the stdout from the
   remote cat starts with the cached password (a lem-webview prompt
   overlay cleanup race), remove it before the content reaches the
   buffer.
  Two problems broke completion:

  1. parse-tramp-path regex "(.+)$" required at least one character after
     the second colon, so "/ssh:host:" (typed before pressing Tab) failed
     to parse.  Changed to "(.*)$" with nil/empty defaulting to "/".

  2. completion-file calls list-directory, which uses uiop:subdirectories
     and uiop:directory-files directly — no virtual filesystem hooks.
     TRAMP paths don't exist locally, so list-directory returned nil and
     the completion candidate list was always empty.

  Fixes:
  - Override *prompt-file-completion-function* with a TRAMP-aware wrapper
    that calls directory-files (which does have virtual hooks) directly.
  - For :sudo, delegate directory listing to the local filesystem (strip
    the "/sudo::" prefix, list locally, re-add the prefix) so completion
    works without triggering a password prompt.
  - Add pathname→string guards in tramp-list-directory-1 and
    tramp-sudo-directory-files (completion-file passes pathname objects
    from merge-pathnames).
  Two bugs in tramp-file-completion:

  1. directory-files was called with the full user input (e.g.
     /sudo::/etc/p) instead of the directory part (/sudo::/etc/),
     causing the delegation to local filesystem to look up a
     non-existent directory path and return nothing.

  2. No filtering was applied to the completion candidates — Tab always
     showed every file in the directory regardless of what the user
     had already typed.  Added case-insensitive prefix matching against
     the partial filename.
@MasterCsquare MasterCsquare changed the title WIP: Tramp for Lem Tramp for Lem Jul 21, 2026
Comment thread extensions/tramp/tramp.lisp
Comment thread extensions/tramp/tramp.lisp Outdated
Comment thread extensions/tramp/tramp.lisp Outdated
Comment thread extensions/tramp/tramp.lisp Outdated
Comment thread extensions/tramp/tramp.lisp
Comment thread src/buffer/file-utils.lisp Outdated
Comment thread src/buffer/file-utils.lisp
@vindarel

Copy link
Copy Markdown
Collaborator

Thanks for the PR!

Can you disclose your LLM usage and methodology?

How long have you been using it, how much tested is it?

Your PR message is useful, can you add it to a README next to the sources?

So the functionality is unix only right? We'll need a feature flag somewhere, to error gracefully for Windows users.

Is the name Tramp appropriate? Since this module isn't a copy, and Lem doesn't aim at being an Emacs copy either. Honest question.

In general I think all the variable and functions names in tramp-… feel too elisp-y. We have packages in CL.


It would be great to have more testers before merging! (I can't test right now for the next few days)

Thanks!

@MasterCsquare

MasterCsquare commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review, I will fix them later.

For LLM, I'm using claude + deepseek. Because deepseek is not the best(compared to chatgpt and Fable, but cheap enough), i have to guide him more, which helps me understanding his code better. LLM is just like a man playing three role at the same time, teacher, friend, slave. Teacher for he known so many things and can explain it in detail. Friend for he can help you do many things. Slave for he can get no tired at all. Almostly my workflow for LLM is simple, i do it step by step to make the codebase larger until the first working version done. After that ,test by myself and fix the bugs. For performance improving , i ask him to analyze if it possible to improve, and check what he said, when everything he said is right, i ask him to implement it.

I only using it for about 2 days, testing is not so complete.

README will be added later.

Right, it only for Unix.

I think Tramp is a suitable name, for nothing related to emacs in this name, and can help the new incoming user to understand what it is faster.

Maybe the LLM read the emacs codes, so it's elisp-y, fix later.

Test is required, currently when input a wrong password, the feedback is not normal, there are still some bugs.

Replace the repeated (or (loop :for f :in *hook* ...) fallback) pattern
with a shared helper. Five functions converted, two left as-is
(file-size has reader-conditional return-from, open-virtual-file returns
multiple values).
@MasterCsquare

MasterCsquare commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

All the Reviews and bugs have been fixed.

Testing is welcome, please report bugs if you found.

There's a typo in the commit messages, so --amend and force push.

The prompt buffer's syntax table treats /, :, @ as symbol chars.
When virtual-path-completions didn't set :start/:end on completion
items, completion-item-range fell back to skip-chars-backward which
skipped the entire TRAMP path (all chars are symbol-constituents),
replacing it with just the filename label.

Fix:
- virtual-path-completions now sets :start (after last /) and :end
  (cursor position) on each completion item, matching the pattern
  used by prompt-file-completion. Only the filename component is
  replaced, preserving the TRAMP prefix.
- Added virtual-directory-namestring for reliable directory extraction
  from TRAMP paths, since SBCL's directory-namestring misparses paths
  like /ssh:host: (treating the host segment as a filename in /).
- Check path-p on expanded instead of input-dir to correctly detect
  TRAMP paths even when directory extraction fails on edge cases.
@MasterCsquare

MasterCsquare commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

I found a new bug, the password prompted(sudo) still insert to the top of buffer, but you can't see it in the buffer but when you cat the file you see it. This should have been fixed before, but it's just in the situation fixing by you can't see.

but I can't reproduce this, it seems to the files opened is opened before in the buggy version, so there is the password in the head, when i open the file i never open before it works as normal.This should be noticed.

The write path (%make-ssh-input-stream) wrote the sudo password and
file content through the same stdin pipe.  'sudo -S' should consume
the first line as the password, leaving the rest for 'cat > file',
but due to stdio buffering the password ended up in the saved file.

This was masked by a read-path workaround in %read-remote-file that
stripped the password from stdout — so Lem's buffer showed clean
content while the password sat on disk at the top of the file.

Fix: pre-authenticate sudo with a one-shot 'sudo -S true' process
whose stdin is closed immediately after the password.  The actual
write command then uses 'sudo -n', so its stdin carries only file
content — no password ever touches that pipe.
@MasterCsquare

MasterCsquare commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

The bug is fixed, can be reproduced by open a new file likeC-x C-f /sudo::/etc/example and input the password for sudo, write something then save,and then cat /etc/example.

@MasterCsquare

MasterCsquare commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

The Terminal is integrated in the new commits.

When a buffer is visiting a TRAMP path, M-x terminal opens a terminal in the remote file's directory on the appropriate host.This works like run eshell in emacs tramp but for lem. Both sudo and ssh are supported.

This make M-x terminal more useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants